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Abstract. A permutation is called layered if it consists of the disjoint union of 
substrings (layers) so that the entries decrease within each layer, and increase between 
the layers. We find the generating function for the number of permutations on 
n letters avoiding (1,2,3) and a layered permutation on k letters. In the most 
interesting case of two layers, the generating function depends only on k and is 
expressed via Chebyshev polynomials of the second kind. 
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1. Introduction and Main Result 

Let 7r C 6 n and r € &k be two permutations. An occurrence of r in tt is a 
subsequence 1 ^ ii < i 2 < ■ ■ ■ < it ^ n such that (jr^ , . . . , TTi k ) is order-isomorphic 
to r; in such a context r is usually called a pattern. We say that tt avoids t, or is r- 
avoiding, if there is no occurrence of r in tt. The set of all r-avoiding permutations 
in & n is denoted 6„(r). For an arbitrary finite collection of patterns T, we say 
that 7T avoids T if tt avoids any r € T; the corresponding subset of & n is denoted 
6„(T). 

In several recent papers [CW, MV, Kr] , the authors express generating functions 
for the number of permutations avoiding certain collections T in terms of Chebyshev 
polynomials. More specifically, the main result of [CW] can be formulated as 
follows. 

Theorem 1.1 ([CW, Theorem 3.1]). Let T ± = {(1, 2, 3), (k - 1, k - 2, . . . , 1, k)}, 
T 2 = {(2, 1,3), (1,2,..., k)}, and T 3 = {(2, 1, 3), (k, 1, 2, . . . , k - 1)}, then the se- 
quences \& n (Ti)\, i = 1,2,3, have the same generating function 

R _ ft-* G&) 

where U r (cos9) — sin(r + l)#/sin# is the Chebyshev polynomial of the second kind. 

Paper [MV] gives a different proof of Theorem 1.1 for the case of T2, and provides 
explicit expressions for the generating functions of permutations avoiding (2, 1,3) 
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and containing (1,2, ... ,k) a fixed number of times, or avoiding (1,2,..., k) and 
containing (2,1,3), via the same Chebyshev polynomials. Still another proof of 
Theorem 1.1 (for all the three cases) based on the use of Dyck paths is given in 
[Kr] , together with the corresponding generalizations (to permutations avoiding the 
shorter pattern and containing the longer one a fixed number of times). 

In this paper, we give a completely different generalization of Theorem 1.1 for the 
case of T\ . Following [Bo] , we say that t <G &k is p-layered if it consists of the disjoint 
union of p substrings (the layers) so that the entries decrease within each layer, 
and increase between the layers. For any k\, . . . , k p ^ 1 we denote by w(k\, . . . , k p ) 
the unique layered pattern of length k = k\ + ■ ■ ■ + k p whose ith layer is of length 
ki. For example, the only 1-layered pattern of length k is w(k) — (k, k — 1, . . . , 1), 
and the only fc-layered pattern of length k is w(l, . . . , 1) = (1,2, ... ,k). Thus, the 
longer pattern in the collection T\ above is just the 2-layered pattern w(k — 1, 1). 

The main result of this paper is as follows. 

Theorem 1.2. Let T = {(l,2,3),r} 7 where t e &u, k ^ p, is an arbitrary p- 
layered pattern, and let Ft(x) — J2 n >a \&n(T)\x n . Then: 

(i) if p = 1 then Ft(x) is a polynomial of degree 2k — 2; 

(ii) if p = 2 then Ft(x) — Rk(x); 

(iii) ifp ^ 3 then F T (x) = (1 - \fl - ±x)j2x. 

In fact, only statement (ii) is new, while (i) and (iii) are added for the sake 
of completeness. Indeed, if p — 1 then r = w(k), and by [ES], there exists a 
permutation in 6„ avoiding both (1,2,..., I) and (to, to — 1, . . . , 1) for n = (to — 
1 ) (Z — 1), and there are no such permutations for any n > (to— l)(l — 1). On the other 
hand, if p > 3 then r evidently contains (1,2, 3), hence |6„(T)| = |6„(1, 2, 3)| = c„, 
the nth Catalan number (see e.g. [Kn, SS]), and the generating function of Catalan 
numbers is known to be (1 — \/l — 4x)/2x. 

Our proof of Theorem 1.2 is based on finding a recursion for the numbers in 
question by purely analytical means. In spite of the paradigm formulated in [Kr], 
that any enumeration problem leading to Chebyshev polynomials is related to Dyck 
paths, it would be tempting to find a proof that exploits such a relation. 

The final version of this paper was written during the second author's (A.V.) 
stay at Max-Planck-Institut fur Mathematik in Bonn, Germany. A.V. wants to 
express his gratitude to MPIM for the support. 

2. Proofs 

Denote by T d ' k the collection {(1, 2, 3), w(k - d, d)}, and by f^ k the number of 
permutations in & n (T d ' k ), 1 ^ d < k — 1. The following observation is obtained 
straightforwardly from the definitions and usual symmetry operations (reversal and 
complement). 

Lemma 2.1. f^ k = f k - ±k for all d, 1 < d < k - 1. 

In what follows we assume that k ^ 3 and d are fixed, and omit the indices d, k 
whenever appropriate; for example, instead of f d ' k we write just /„. Moreover, it 
follows from Lemma 2.1 that we may assume that 



k — d > d. 



(1) 
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For any n > 1 and any m such that 1 ^ m < n, we denote by g n (ii, ■ . . , i m ) = 
gn' k (ii, ■ ■ ■ ,i m ) the number of permutations n G & n (T d - k ) such that wj — ij for 
j = 1, . . . , m; the corresponding subset of & n (T d - k ) is denoted by G„(ii, . . . , i m ). 

The following properties of the numbers g n {h, ■ ■ ■ , « m ) can be deduced easily 
from the definitions. 

Lemma 2.2. (i) Let n ^ 2 and 1 ^ i < n, then 

g n (. . . .. ,i, . ..) = 0. 

(ii) Lef n > 3, l^m^n — 2, and n — 2 ^ i\ > ii > • • • > i m > 1, i/ien 

g„(ii,...,i m ,j) = /or i m + 1 <j < n- 1. 

(iii) Let n ^ 3, 1 < m ^ min{n, k — d} — 1, and n — 1 ^ i\ > ii > • • • > i m > 1, 

ff„(ii, . . . ,i m ,n) = ff n -i(ii, • • . ,i m ). 

(iv) Lef n > 3, 2 < m < n, ii > n — d+ 1, and n > ii > 12 > • • • > i m ^ 1; then 

g n {k, ■ ■ ■ ,im) = .9ri-l(«2, • ■ • ,i m )- 

(v) Let n > k, k — d^m^n — d, and n — d > i\ > ii > • • • > i m ^ 1> i/ien 

5„(ii, . . . ,i m ) = 0. 

Proof. Property (i) is evident. To prove (ii) is enough to observe that if 7r € 
G n (ii, . . . , i m , j) then the entries i m , j, and n give an occurrence of (1, 2, 3) in n, a 
contradiction. 

To prove (iii), denote by 7r* the permutation obtained from tt by deleting its 
largest entry. Then ir G G n (ii, . . . , i m , n) if and only if 7r* G G„_i(ii, . . . , i m ), since 
entry n placed as in (iii) cannot be used in an occurrence of (1, 2, 3) or w(k — d, d) 
in tt. 

The proof of (iv) goes along the same lines. We denote by tt° the permutation 
obtained from it by deleting entry i\ and subtracting 1 from all the entries greater 
than ii; then tt € G n (i\, . . . , i m ) if and only if tt° G G„_i(i2, . . . , i TO ). Indeed, the 
only if part is trivial, so assume that tt° G G„_i(i2, . . . ,i m ), and there exists an 
occurrence of (1,2,3) or w(k — d,d) in ir. Evidently, such an occurrence makes 
use of i\. If («i, ji, J2) is an occurrence of (1,2,3), then j\ and ji he to the right 
of i m , and since i\ > i m , the triple (i m ,ji,ji) is an occurrence of (1,2,3) in tt° , 
a contradiction. Otherwise, i\ enters an occurrence of w(k — d,d), and hence tt 
contains at least d entries strictly greater than ii, a contradiction. 

Finally, to prove (v), consider all the entries of tt greater than i\. There are 
at least d such entries, and all of them lie to the right of i m . If a pair (ji,ji) of 
these entries is an occurrence of (1, 2), then (i m , j\, ji) is an occurrence of (1, 2, 3), 
a contradiction. Hence, all these entries provide an occurrence of (p,p — 1, . . . , 1) 
for some p > d; therefore, any d out of these entries, together with any k — d entries 
out of i\, . . . , i m , give an occurrence of w(k — d,d). □ 
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Now we introduce several objects that play an important role in the proof of the 
main result. For n ^ d + 1 and 1 ^ m ^ min{n, k} — d we define 

l(n,m) = {(ii, . . . ,i m ): m- j < ij- - 1, j = 0, . . . ,m - 1}, i = n - d + 1, 

J(n,m) = {(h, . . . ,i m ): m- j < ^ ij- - 1, j = 0, . . . ,m - 1}, i = n - d + 2, 
K(n,m) ={(i 1 ,...,i m ): (i lt .. .,i m -i) & l(n,m- 1),1 < i m < z m _i - 1}, 

and put 

A(n,m) = ^ 5f n (zi, . . . ,i m ). 

(ii . . . ..i m ) Gl(n,m) 

Theorem 2.3. Let n > d + 2 and 2 ^ m ^ min{n, fc — d} — 1, i/ien 
A(n, m) = i4(n - 1, m - 1) + m + 1). 

Proof. By the definition, 

n 

A(n,m)= ^2 ^2/9n(h,---,i m ,j)- 

(ii,...,i m )£l(n,m) j=l 

Let us decompose the internal sum into three subsums over the intervals 1 ^ j ^ 
i m — 1, i m ^ j ^ n — 1, j = n, and denote them Si, S2, and 6*3, respectively. 
Evidently, 

Si = ^2 9n(il, ■ ■ -,im,j) 

(i 1 ,...,i m )eT(n,m) 0=1 

= X] 9n(h,---,im,j)- 

(ii,...,i m ,j)e/C(n,m+l) 

ft is easy to see that if (i\, . . . ,i m , j) e )C(n, m+ 1) and (i\, . . . ,i m ,j) ^ J(n, m + 1) 
then there exists p, 1 < p < m, such that i p = j, and therefore by Lemma 2.2(i), 

5i = X g n (ii,---,i m ,j) = A(n,m + 1). 

(ii,...,i m ,j)eX(n,m+l) 

Next, 

n-l 

#2 = X ^2 9n(k,---,i m ,j) = 

(ii,...,i m )el(n,m) j=»m 

by Lemma 2.2(i) and (ii). Finally, by Lemma 2.2(iii), 

5*3= X 5n(»l, • • • ,i m ,n) 

= X ffn-l(«l, • • • ,im)- 

(ii,...,i m )ej(n-l,m) 
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Denote the latter expression by B(n — 1, to), so 

A(n,m) = A(n,m+ 1) + B(n- l,m), (2) 

for n ^ d + 1, 1 ^ to ^ min{n, fc — d} — 1. 
Consider now m + 1). One has 

B(n,m + 1)= ^ g n {ii,...,i m ,j) 

(ii,...,i m ,j)£l(n,m+l) 

+ X! 9n(n-d+l,i 2 ,---,im,j), 

(i 2 -l,...,*m-l,j-l)€j'(n,m) 

and by Lemma 2.2(iv), the second term in the right hand side equals B(n— l,m). 
Therefore, 

-B(n, m + 1) = m + 1) + £>(n — 1, m) 

for n > d + 1, 1 ^ to < min{n, fc — d} — 1. 

Comparing this with (2) one gets B(n, m+ 1) = m) for n > d + 1, 1 ^ to < 
min{n, k — d} — 1, and hence 

m) = A(n - 1, to - 1) + A(n, to + 1) 

for n ^ d + 2, 2 ^ to ^ min{n, fc — d} — 1. □ 

Our next aim is to express A(n, to) in terms of fj's. We start from the following 
result. 

Lemma 2.4. Let n ^ d + 2 trod 1 < j < d, then 

g n (n + 1 - j) = (° ~ 1 ~ *) /„_!_<. (3) 

4 =o \ * / 

Proof. Let n > 3; it is clear that g„(n) = / n -i, hence (3) holds for j = 1 and n ^ 3. 
Moreover, if d 7^ 1 then g n (n — 1) = / n _i as well, and hence (3) holds also for j = 2 
and n ^ 3. This gives the basis for the induction. 
Next, observe that 

g n (n + 2 - j) - 5„(n + 1 - j) = g n -i(n + 2 - j) 

for n ^ 3, 3 ^ j ^ min{d, n). Indeed, 

n 

9n{r) = ^g n {r,i)- 
i=i 

The sum in the right hand side is subdivided into four sums: for 1 ^ i ^ r — 1, for 
i = r, for r + 1 ^ i < n — 1, and for i = n, respectively. Using Lemma 2.2(iv), (i), 
(ii), and (hi) respectively, we get 

r 

9n{r) = ^2g n -i(i) 
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for n ^ 3 and n + 1 — d ^ r ^ n — 1. Observe that condition m k — d — 1 
from Lemma 2.2(iii) is satisfied by virtue of (1) and k > 3. It remains to substitute 
r = n — j + f and r = n — j + 2, and to find the difference between the right hand 
sides; the restrictions on r are transformed into 3 ^ j ^ d. 
Finally, it is easy to check that 

g<-<rv»-g<-'>fT>~ 

for any n 2 and j 3, and hence (3) is satisfied for n 3 and 1 < j < 
min{d, n — f}. It follows now from (I) that (3) is satisfied for n ^ d + 2 and 
1 < j < d. □ 

We are now ready to find an expression for A(n, m) similar to expression (3) for 
g n {n+ 1 - j). 

Theorem 2.5. Let k and 1 ^ m < fc — d, i/ien 

A(n, TO ) = f;Virf TO+ .^ i )/— w 

Proof. Similarly to the proof of Lemma 2.4, we first check (4) for m = 1 and m = 2. 
Clearly, 

^("i !) = XI 5n0') = fn - £ 9n{n + 1 - j). 

j=l i=0 ^ ' 

d— 1 d / • 1 -\ 

/» - E(- l )v.-i- i E r " , ) 

i=0 j=i+l ^ ' 

for n ^ d + 2, so (4) holds for m = 1 and n ^ d + 2. 
Next, by (2), we have 



Using (3) we get 

A(n,l) = 



A(n, 2) = A(n, 1) - B(n — 1, 1) = A(n, I) - A(n - 1, 1) - g n -i(n - d) 
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for n ^ d + 2 (condition to = 1 ^ fc — d — 1 follows from (1) and k ^ 3 as in the 
proof of Lemma 2.4). Using (3) for j — d and (4) for m — 1 we get 



A(n,2) = J2(-iy( d+1 'V 

i=0 ^ ' 



2-i 



i=0 x 7 i=0 

d+2 

/ < / 4- V / \ 

fn- 

\ / / 

for n ^ d + 2, so (4) holds for m = 2 and n ^ <i + 2. 
Finally, by Theorem 2.3, 



A(n, m) = A(n, m - 1) - A(n - 1, m - 2) 
for n ^ d + 2 and 3 ^ to ^ fc — d, so by induction we get 



2 = v 7 i=0 



=E(-D'( ra+ r>-. 

for n ^ d + 2 and 1 < to ^ mm{n + 1, k} — d, and hence for n ^ k and 1 ^ m < 
fc-d. □ 

Now we are ready to prove Theorem 1.2 for k ^ 3. First of all, it follows from 
Lemma 2.2(v) that A(n, k — d) = for all k. Hence, by Theorem 2.5, 

k — i 



i=0 

for n ^ fc, or, equivalently, 



EEt-^YK^""^ - 

•'— ri>fe 



Evidently, for n ^ k — 1 the layered restriction is void, so by [Kn, SS], /„ = c„ 
for < n < fc — 1, where c„ is the nth Catalan number. Therefore 



£(-*)'(* l ) [F T ( X )- k ^2^cA =0. (5) 
i=o v ' \ j=o J 

Recall that the Chebyshev polynomials of the second kind satisfy relation 



I 
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(see [Ri, pp. 75,76]), while the Catalan numbers satisfy relation 

£(-i)f:V* = (-i)'( 



I 



for I ^ k — 1 (see [Ri, pp. 152-154]), and hence 
k (h i\ k - i - 1 k - 1 1 / u A fe_1 (k 1 7\ 

Therefore, (5) yields 
for k > 3. 

The remaining case fc = 2 is trivial, since #2(2) = (1— x) _1 , which is equivalent to 
the fact that there is exactly one permutation of length n avoiding w(l, 1) = (1, 2). 
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